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(5) consensus on components of and opinions on the quality of the 
validated job profile? and (6) evaluation of the validation approach. 
Findings indicated that the validation approach was satisfying. 
Although there was a considerable amount of prior agreement with the 
draft job profile components, the validation sessions created an even 
greater agreement with the final job profile components. The 
validation approach was also satisfying with regard to product 
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showed that process and product quality were significantly related. 
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introduction 



The study is part of a project . the Dutch Institute for Curriculum Development. The mam 
objective of the project Is to create and validate job profiles for three health care education 
programs on dietetics, pottotherapy, and activity therapy. These job profiles were meant for 
curriculum development and revision. Validation of job profiles for curriculum profiles is a key 
component in curriculum development and revision as all stakeholders in the educational 
program have to approve the job prof Be because this serves as the common basis for further 
curriculum development and revision. Acceptation of the final curriculum is largely depending 
ofi approval of the job profiles. 

Choosing the appropriate validation strategy is therefore a critical issue in job profile 
validation for nationally based curriculum development, and of significant importance for aU 
those who are engaged in the devetopent process. There are several strategies available, but 
empirical results of the effects of these strategies are largely unknown. 

The objectives of this study are: 

- to evaluate the approach that is employed for validation of job profiles for curriculum 
development in three health care programs; 

- to evaluate the quality of the products that are validated; 

- to test the relation between the process quality of the validation approach on the one hand 
and product quality of the job profiles that are validated on the other hand. 

Theoretical framework 

Job profile validator for curriculum development or curriculum revision is an emerging 
component in curriculum development for vocational and professional education and training, 
in the sixties the classical approach was to perform large job surveys in order to determine 
the commonalities between job tasks in related job groups. Common tasks were grouped 
according to the results of cluster analyses and the curriculum was designed according to the 
structure of these task clusters. Examples of this approach can oe found with Smith (1973; 
1974) and Randhawa (1978) 

In more recent approaches for curriculum development and revison for vocational and 
professional education and training, results of these job surveys play a more modest part as 
multiple perspectives on the data and the results can lead to divergent interpretations and 
consequently to totally different curricula. Furthermore, curriculum theoretical contributions of 
Tyler (1950), Taba (1962), Schwab (1971), Goodtad (1979), Tanner & Tanner (1980), 



Beauchamp (1981), Walker (1990), and many others have indicated that curriculum 
development processes involve multiple data and justification resources. Therefore we 
contend that mixes of data collection strategies and heterogeneous group interpretations 
have to be employed to enhance the quality of the vaBdatton process. This cafe for 
delfoeratfve curriculum development in which all stakeholders are involved. They reflect on 
the current state of the art in the respective curriculum domain, and try to establish agreement 
on the curriculum that It at stake. 

In this study we evaluated the approach that is employed for validation of three job profiles, 
the quality of these job profiles, and tite relation between process and product quality. A job 
profile can be defined as the structured set of duties and tasks that serves as a reference 
guide for curriculum development and revision in vocational and professional education and 
training. The validation approach we used is a practical application of a small group stategy, 
described earlier by Frey (1982), Nljhof & Mulder (1989), Mulder & Thijsen (1990), Mulder & 
Te Brake (1990) and Mulder (1989; 1991). The approach consists of three parts: information, 
deliberation and consensus (see figure 1). Information is the result of an empirical front-end 
analysis that is documentented in a so-called information document, which is needed for 
prudent preparation of the validation session. Deliberation is an approach for decision making 
described by Schwab (1971) and Walker (1975) and many others. In our project it is aimed at 
establishing consensus on the job profiles. 

Figure 1 about here 

Professionals from the trade and the educational institutions for which the job profile is 
validated are invited for the validation meeting. The teascn for this is that job profiles are 
interpreted differently from both sides, and clear understanding of these differences is crucial 
for the validation process. 

information provision for those engaged in the validation process has to be divergent, and 
conflicting sections, that may lead to different conclusions as to wether what will be the 
itent and structure of the validated job profile, have to be highlighted to prevent the 
participants from reaching cosmetic consensus. Deliberation has to be focussed on sharing 
perspectives, opinions and arguments for and against given proposals for the job profile. This 
interaction process should lead to consensus on the job profile. 



Data source, methods and techniques. 

Data Is used from three cases In which a job profile is approved, in these cases 12, 10 and 11 
persons were present. They were either with health care institutions or with educational 
institutions. 

Data is coflee*ed with six questionnaires on: 

1. Personal characteristics, motives and expectations, the information document and the 
draft job profile. 

2. Opinions on issues and expected consensus. 

3. Communication rules. 

4. The decision making process. 

5. Consensus on components of and opinions on the quality of the validated job profile. 

6. EvaJbatton of the validation approach. 

Questionnaires 1 and 2 were completed some days before the validations sessions, 
questionnaire 3 at the beginning of the first session, 4 during the last session, and 5 and 6 
within a week after the validation sessions. 

The deliberative validation approach is evaluated for the amount of convergence of the 
opinions of participants on the six different components of the job profile. These components 
are: 

-a brief description of the job; 

- a description of task domains; 

- a description of the basic structure of the task domains; 

- an overview of (task domains) and job tasks; 

- a desription of trends and developments; 

- a match betwten job tasks and occupational sectors. 

These components are evaluated twice. Once prior to the validation sessions, and once after 
these sessions. Participants are asked to rate the level of prior agreement with the job profile 
components on a five point scale (1 «= strongly agree; 5 - strongly disagree), and to rate the 
level of group agreement established during the sessions on a seven point scale (1 » 
unanimity ; 7 = Insurmountable controversy). To enable comparability between both 
measurement moments the two extreme positive and negative values of the second scale are 
receded to one score, so scores 1 and 2 were taken together, as well as scores 6 and 7. Thus 
we created a five point scale, of perceived group agreement with the job profile that is 
validated during the sessions, for the seond measurement moment. The shift in agreement 
with the job profile components is determined by computing the means of the a priori and a 
posteriori agreement scores on the job profile components, as well as the mean difference 



between the score moments. The magnitude of this shift shows the impact of the validation 
sessions on the job profile agreement. 

To evaluate the quality of the products that are validated participants are asked to rate the 
components of the job profile on a seven point scale. This scats is converted to a five point 
scale by merging both two extreme values of the scale as described above. Means are 
computed for the whole job profile, as wed as for the seperate components. The difference 
between the perceived product quality of the different cases is tested by a Kruskall-WaBis 1- 
wayAnovatest. 

To test the process quality of the validation sessions questionnaire 4 on the decision malting 
process is used. This questionnaire is developed and extensively described by Frey, MaUiou, 
Langeheine & 'orton-Kruger (1988). Suffices a here to say that this questionnaire consists of 
two parts; the first part is used in this study. It contains 32 Lfkert items, that represent 8 
criteria of process quality. Each criterion is operationaiized into four items, two of wich are 
formulated positively, and two negatively, whereas two required external observation, and two 
seif -observation. The items are randomly Distributed in the questionnaire. The negatively 
formulated Kerns were receded for data processing. Means for all participants, and by 
validation session are computed. A Kruskafl-Waliis 1-way Anova test is performed on these 
means to test the differences of process quality between the three cases. 
To test the relation between the process quality of the validation approach on the one hand 
and the perceived product quality of, and the agreement on the job profiles that are validated 
on the other hand, the mean process quality score is computed, as well at; the mean quality 
and agreement score. These scores are ranked across the participants, and a Spearman 
rank-correlation test is performed. 

FinaHy we performed a Spearman rank-correlation test between the perceived product 
agreement on the one hand with perceived product quality, individual prior agreement (with 
the drali job profile), and expected product agreement on the other hand. 

Results. 

Table 1 shows the results for the first research question. The mean prior agreement with the 
job profile components is 2.40 (sd = .45), the mem post agreement is 1.43 (sd « .48). The 
mean difference between the a priori and a posteriori agreement is 1 .03 (sd » .40). Due to 
partial non-response this total mean difference is based on 25 of the 33 cases, which caused 
a minor deviance between the difference of the means of the pre and postscores on the one 
hand and the total mean difference on the other hand. 

This means that the mean shift of opinions on job profile components due to the validation 
sessions is 1 .03 point on the five point scale. 



if we took at the different job profile components, we see a range of 2.09 (component 3) to 
2.71 (component 1) for the pre-consensus scores, whereas we see a range of 1.24 
(component 3) to 1.72 (component 6) for the post-consensus scores. The mean difference 
ranges from 1 .32 (component 1) to .76 (component 5). 

M we divide the range of the five point scale In five subranges of 1.0 - 1.5 (+♦), 1.5 - 2.5 (+), 
2.5 - 3.5 (+/-), 3.5 - 4.5 (-). and 4.5 to 5.0 (~) 'or very positive, positive, neutral, negative, and 
very negative mean group scores, we see that a priori three job profile components were 
evaluated in the positive, and three in the neutral subrange, whereas a posteriori four 
components were evaluated in the very positive, and two in the positive subrange. 

Table 2 about here. 

The data snow (see table 2) that the mean perceived quality of the job profile Is 1.51 (sd ° 
.57} on a scale ranging from 1 (» positive) to 5 ("negative). The mean perceived quality by job 
profile component vanes from 1.33 for component 2 to 1.79 for component 6. if we again 
divide the range of the five point scale in five subranges that vary from very positive to very 
negative, we see that the mean scores of four components are falling in the very positive 
subrange, whereas two of the mean scores fall in the positive subrange. 

Table 3 about here. 

Table 3 shows that for case 1 the mean perceived product quality is 1 .43 (sd = .50), for case 
2 1.07 (sd = .16), and for case 3 1.97 (sd = .53). A Kruskall-Wallis 1-way Anova test of 
perceived product quality by case shows a Chi-Square of 16.47 (corrected for ties) (p - 
.0003). 

Table 4 about here. 

Table 4 shows the mean process quality of the validation sessions as determined with 
questionnaire 4. Accross the cases the mean process quality is 2,39 (sd ■ .22) on a scale 
ranging from 1 (= very positive) to 5 (= very negative). If we again take 2.50 as the 
demarcation point for the positive and negative subrange, we see that the total mean process 
quality falls In the positive subrange. The means by validation session (case 1 to 3) vary from 
2.2 (case 2) to 2.6 (case 3). Cases 2 and 1 fall in the positive subrange, whereas case 3 falls 
in the neutral subrange. A Kruskall-Wallis 1-way Anova test showed a Chi-Square of 10.43 
(corrected forties, p * .0054). 



The Spearman rank correlation coefficients for process quality and perceived product quality 
ofthe job profile is. S5 (p= .000) and for process quality and agreement on the job profile this 
is.57(p».001). 

Table 5 about here. 

Table 5 shows rank correlations coefficients and significances of perceived group agreement 
with the final job profile (product agreement) on the one hand by perceived quality of the final 
job profile (product quality), individual agreement with the draft Job prof Be (draft agreement), 
and expected agreement with the final job profile on the other hand. The correiatior. between 
product agreement and product quality is .66 (p » .000), between product agreement and 
draft agreement .66 (p =» .000), and between product agreement and expected agreement .55 
(P-.003). 

Conclusions. 

If we return to the objectives of this study, we can conclude that the validation approach is 
satisfying. Although there is a considerable amount of prior agreement with the draft job 
prof Be components, the validation sessions created an even greater agreement with the final 
job profile components. The mean gain in agreement appeared to be quite substantial: 1 .03 
points on a five point scale. The gain might have been greater even when there had been less 
prior agreement with the draft job profile components. 

If we took at product quality of the job profile, the validation approach is also satisfying. 
Although the perceived quality of the job profile components varied significantly across the 
three cases (p « .0003). the mean quality appeared to be 1.51 on a five point scale, of which 
10 is the maximum positive extreme. The variation between cases is 1 .07 for case 2, and 
1 .97 for case 3. This implies a difference in variation of .90, but all means of the cases fall in 
the positive subrange of the scale, which in fact is also very satisfying. 
The process quality of the validation approach is scored less positive; the mean across the 
cases is 239 on a five point scale (1 » positive extreme). There is a significant difference of 
process quality between the cases (p » .0054;, cases 1 and 2 fall in the positive subrange, 
whereas case 3 is scored relatively toss positive as it falls in the neutral subrange. We can 
conclude that the overall process quality across the cases is satisfying, but that case three 
scored below average. On the other hand the mean process quality score of cases 3 is 2.6 on 
the five point scale, which in fact is nearly positive, and far from negative (the demarcation 
point for negative means is 35). 



The results show significant rank correlation coefficients between process quality and 
perceived product quality of the job prof He (r$» .65; p= .000), and agreement on the product, 
the final job profile that is approved during the validation sessions (r 8 » .57; p - .001). This 
result confirms the study by Frey, Frei & Langehgine (1989) that curriculum development 
processes affect the resulting products, but also shows that there is a wfthin-vafidatton- 
apprcach variety of process quality. 

RnaDy the results show that the rank correlation coefficients between the perceived group 
agreement with the final job prof Re on the one hand, and the perceived quality of the ?inai job 
profile (r s « .86), the individual agreement with the draft job profile (r s = .66), and the a priori 
expected agreement of the group after the validation sessions (r s ° .55), all are significant. 
On the basis of the results for the three cases we studied, we can conclude that process 
quality seems to be important for product quality, which in turn seems to be important for 
agreement with the final product. And if there is a considerable amount of preconsensus on 
the draft job profile, participants accept K as a valuable starting document, that can even be 
improved during the validation sessions, which aJso results in significant rank correlations 
between the agreement with the final product, and the agreement with the draft job profile, as 
welt as with the expected agreement with the final job profile. 

Discussion 

Validated job profiles are important for curriculum content justification in vocational and 
professional education and training. Validation approaches are manifold, but effects of these 
approaches are hardly studied. This causes serious problems in nationally based curriculum 
development and revision in vocational and professional education and training. The pertinent 
question is: "What approaches result Hi valid job profiles?* This study shows that the 
approach that is used does result in valid job profiles. In this sense it is a contribution to 
curriculum theory for vocational and professional education and training. 
The results showed that prior to the validation sessions, a considerable amount of 
preconsensus existed already. It is important to evaluate the level of preconsensus on draft 
products, as this is an important condition as to whether the validation sessions are 
necessary or not. But on the basis of this study we cannot conclude that positive effects also 
hold when there is only limited preconsensus or even controversy with respect to the job 
profile. 

The study snows that process and product quality are significantly related, which implies that 
attention for validation processes is justified. More research is needed to test the relative 
weight of the components within the approach and contextual factors to explain the variation 
in quality of both processes and products 




In future studies participation effects nave to be controlled by having the products of validation 
sessions evaluated by independent raters who were not Involved in the validation process. 
In future studies follow-up processes should also be analyzed: how were the Job profiles 
interpreted by users? How were they used? With what results? How did they enhance (or 
perhaps frustrate) the curriculum development or revision process? Do job profiles control the 
content of tests? 
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Figure 1 Components of the validation approach 
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Table 1 Mean scores on the individual (Ind) agreement (agreem) prior to the validation 
sessions and perceived group agreement on components (comp) of the job profiles after the 
validation sessions (range 1 ■ positive, 5 * negative), mean difference scores (D1 to D6), and 
total mean difference (MDtotaJ). 



Variable 



Mean 



Std Dev 



N Label 



MPREAGR 
MPOSAGR 
MDTotal 



2.40 
1.43 
1.03 



.45 26 Mean pre agreement 
.48 32 Mean post agreement 
.40 25 Mean total difference 



vniin 

VR1AV 
Dl 



2.71 
1.39 
1.32 



.90 31 Ind agreem comp 1 
.70 33 Group agreem comp 1 
1.01 31 Difference comp 1 



VR21I1 

VR2AV 

D2 



2.33 
1.27 
1.06 



.65 33 Ind agreem comp 2 
67 33 Group agreem comp 2 
97 33 Difference comp 2 



VR31II 

VR3AV 

D3 



2.09 
1.24 
.88 



.39 32 Ind agreem comp 3 
.44 33 Group agreem comp 3 
.49 32 Difference comp 3 



VR41II 

VR4AV 

D4 



2.55 
1.33 
1.26 



68 31 Ind agreem comp 4 
65 33 Group agreem romp 4 
.77 31 Difference comp 4 



VR51II 
VR5AV 

D5 



2.30 
1 .55 
.76 



.59 33 Ind agreem comp 5 
.79 33 Group agreem comp 5 
.90 33 Difference comp 5 



VR61II 

VR6AV 

D6 



2.52 
1 .72 
.79 



.69 2<» Ind agreem comp 6 
.96 32 Group agroem comp 6 
. ?2 28 Difference comp 6 
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Table 2 Mean quality perceptions (qua!) by job profile component (comp). 



Variable 



Mean 



St J Dev 



N Label 



Total 



1.51 



.57 31 Total qual 



VR1BV 
VR2BV 
VR3BV 
VR4BV 
VR5BV 
VR6BV 



1.36 
1.33 
1.36 
1.36 
1.-.7 
1.79 



.74 33 Qual comp 1 

. 69 33 Qual comp 2 

.65 33 Qual comp 3 

.60 33 Qual comp 4 

.92 31 Qual comp 5 

1-08 33 Qual comp 6 
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Table 3. Mean quality perceptions of job profile by cases. 



Variable Value Label Mean Std Dev Cases 

For Entire Population 1.5054 .5668 31 

CONF case 1 1.4323 .5043 10 

CONF case 2 1.0667 .1610 10 

CONF case 3 1.9697 .5261 11 
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Table 4. Means on process quality (1 » very positive; 5 ■ very negative) by cases. 



Variable 


value JjaOeX 


wean 


Std Dev 


Cases 


For Entire 


Population 


2,3895 


.2220 


28 


CONF 


case 1 


2.4403 


.2080 


11 


CONF 


case 2 


2.2168 


.1458 


10 


CONF 


case 3 


2.5536 


.1820 


7 
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Table 5. Rank correlations coefficients and significa.ices of perceived group agreement with 
tho final fob profile (RPRODAGR) by perceived quality of the final job profile (RPRODQUAL), 
individual agreement with draft job profile (R1NDAGR), and expected agreement on the final 
job profile (REXPAGR). 

Correlations: RPRODQUA RINDAGR REXPAGR 
RPRODAGR .8572 .6583 .5455 



24) 



24) 



24) 



P 



.000 



.000 



p 



.003 




(Coefficient 



/ (Cases) / 1-tailed Significance) 
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